Method and apparatus for cooling an equipment enclosure through closed-loop, liquid-assisted air cooling in combination with direct liquid cooling

ABSTRACT

A method and an apparatus for cooling, preferably within an enclosure, a diversity of heat-generating components, with at least some of the components having high-power densities and others having low-power densities. Heat generated by the essentially relatively few high-power-density components, such as microprocessor chips for example, is removed by direct liquid cooling, whereas heat generated by the more numerous low-power or low-watt-density components, such as memory chips for example, is removed by liquid-assisted air cooling in the form of a closed loop comprising a plurality of heating and cooling zones that alternate along the air path.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of the filing date of U.S.Provisional Patent Application No. 60/695,378; filed on Jun. 30, 2005;the disclosure of which is incorporated herein in its entirety.

The present application was made with U.S. Government support underContract No. NBCH 3039004 awarded by the Defense, Advanced ResearchProjects Agency, in view of which the U.S. Government has certain rightsin this invention.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the cooling of enclosures, such asracks, for diverse types of equipment, such as heat-producingelectronics, through a combination of air and liquid cooling for veryhigh total power levels of the equipment.

For instance, as heat is generated during operation of electronicequipment, such as that comprising an integrated-circuit chip (IC), thethermal resistance between chip junction and the medium employed for theremoval of heat must be sufficiently small in order to provide ajunction temperature that is low enough to insure the continued reliableoperation of the equipment. However, the problem of adequate heatremoval becomes ever more difficult to solve as chip geometry is scaleddown and operating speeds of the electronic equipment are increased,resulting in an increased power density (W/cm²) at the surface of thechip. The problem is further exacerbated when different types of chipsin close proximity with each other possess different coolingrequirements. For example, in a computer system, a processor chip mayhave a much higher power density than closely located memory chips.Furthermore, as another example, different types of chips may havedifferent maximum-allowable junction temperatures. Such coolingrequirements impose mechanical and thermal packaging challenges to theequipment design that can limit the performance thereof. In the currenttechnology, the power density of processor and other kinds ofhigh-performance chips is rapidly approaching levels that exceed thecapability of forced-air cooling, necessitating the use of liquidcooling for some applications and installations in order to be able toattain the requisite degree of cooling for the equipment.

The cooling of computer racks and other types of electronic equipment istypically accomplished by forced-air cooling; however, liquid-assistedair cooling and direct-liquid cooling, frequently with water as thecooling medium, have also been widely employed. This concept isdiscussed in Richard C. Chu, et al., “Review of Cooling Technologies forComputer Products”, IEEE Transactions on Device and MaterialsReliability, Vol. 4, No. 4, pp. 568-585, (December 2004). Inliquid-assisted air cooling, a liquid-cooled heat exchanger is placed ina heated air stream in order to extract heat and reduce the airtemperature before it is expelled into the room. Chu, et al. (Supra)also describe the problems encountered with data-center thermalmanagement, in which the power dissipated for each equipment rack isapproaching 30 kW. In a typical modern data center, water-cooledair-conditioning units or other external cooling devices are used toprovide, through perforations in a raised floor or through ducts, astream of chilled air to the computers, in which the air is heated, anddownstream of which the air is returned to air-conditioning units so asto be chilled again. Significant problems encountered with this approachinclude the need for the large circulatory volume of air required toadequately cool the electronics, the extensive raised-floor spacerequired to handle this air volume, the accompanying high acoustic noiselevels encountered in the room, and the difficulty of controlling theairflow in the room to prevent already-hot air from re-circulating intothe electronics, thereby potentially leading to overheating andelectronic failure of the equipment. Moreover, the computer machine roomcan be uncomfortable for human occupancy because of large temperaturedifferences present between room areas cooled by the cold inlet air androom areas heated by the hot outlet air. It is noted hereby thattraditional data-center cooling is basically quite similar toliquid-assisted air cooling in that, in both instances, heat isinitially transferred from the electronics to air. The differenceresides in the location of the subsequent heat transfer from air toliquid: in traditional data-center cooling, this air-to-liquid heattransfer occurs outside the computer racks, typically inair-conditioning units where the liquid is water, whereas inliquid-assisted air cooling, it occurs within the computer racks.

2. Discussion of the Prior Art

Various methods and apparatus have been developed in the technology forthe purpose of imparting adequate cooling to diverse types of operatingequipment, such as electronic devices functioning at high power levelsand which generate considerable amounts of heat, which must bedissipated.

Chu, et al., U.S. Pat. No. 6,819,563 B1, which is commonly assigned tothe present assignee, and the disclosure of which is incorporated hereinby reference, discloses a method and system for augmenting the aircooling of rack-mounted electronics systems by using a cooling fluid tocool air entering the system, and to remove a portion of the heatdissipated by the electronics being cooled. A drawback in the adding ofheat exchangers to an electronic rack is due to an increased flowresistance that reduces airflow through the rack. In the patent, the airpath is an open loop, whereas the present invention is directed to theprovision of a closed-loop air path inside the rack.

Chu, et al., U.S. Pat. No. 6,775,137 B2, which is commonly assigned tothe present assignee, and the disclosure of which is incorporated hereinby reference, relates to an enclosure apparatus that provides for acombined air and liquid cooling of rack-mounted, stacked electroniccomponents. A beat exchanger is mounted on the side of the stackedelectronic components, and air flows from the front to the back withinthe enclosure, impelled by air-moving devices mounted behind theelectronic components. A drawback in adding a heat exchanger to the sideof an electronics rack is the requirement for an increase in floorspace. Moreover, a front-to-back airflow within the confines of the rackdoes not allow for the use of a continuous midplane for the electroniccomponents.

Patel, et al, U.S. Pat. No. 6,628,520 describes an apparatus for housingelectronic components that includes an enclosure, mounting boards withelectronic components mounted thereon, a supply plenum for cooling air,one or more outlets, which are directed toward the mounting boards, oneor more heat-exchanging devices, and one or more blowers. A significantlimitation in this arrangement resides in that inlet and outlet plenumsfor the air are needed along opposite sides of the electronics rack, inaddition to the space required at the top and bottom of the rack, whichis used to reverse the direction of the air flow.

Ishimine, et al., U.S. Pat. No. 6,621,707 pertains to an electronicapparatus comprising a motherboard, multi-chip modules mounted to themotherboard, cooling members for cooling the multi-chip modules, arefrigeration unit for cooling the cooling members to room temperatureor lower, and a connection structure to releasably couple eachmulti-chip module thermally and mechanically to the refrigeration unit.In contrast therewith, the present invention does not use arefrigeration unit, or require a substantially hermetically sealed boxstructure, or a drying means for supplying dry air for cooling purposes.

Sharp, et al, U.S. Pat. Nos. 6,506,111 B2 and 6,652,373 B2 each describea rack with a closed-loop airflow and a heat exchanger. The air flowsvertically up one side of the rack, horizontally across electronicsdevices, vertically down the other side of the rack, and then across aheat exchanger located in the base, or optionally on the tops, of therack. Because the air path is much shorter for memory cards near theheat exchanger location, a perforated plate is included in one of thevertical paths to enable adjustment of the airflow across the variousmemory cards to match some desired, e.g., constant distribution. Thepresent invention does not require the vertical plenums, which occupyvaluable space, or the perforated plate.

Parish IV, et al., U.S. Pat. No. 6,462,949 B1 discloses a coolingapparatus using “low-profile extrusions” to cool electronic componentsmounted on a board or card, whereas contrastingly, the present inventiondoes not use any “low-profile extrusions” or similar structure.

Miller, et al., U.S. Pat. No. 6,305,180 B1 discloses a system forcooling electronic equipment using a chiller unit between adjacent racksfor returning cooled air to ambient Contrastingly, the present inventionis distinct from the foregoing because in the system described therein,the air is re-circulated within the rack rather than being expelled tothe ambient environment.

Go, et al., U.S. Pat. No. 5,144,531 is directed to a liquid coolingsystem comprising cold plates attached to their respective circuitmodules, quick couplers for connecting flexible hoses to these coldplates, a supply duct, and a return duct to form strings of cold plates,which are connected between the supply duct and the return duct. Valvedquick couplers are used for the connection to the supply duct and thereturn duct, and valveless quick couplers are used for the connection tothe cold plates. In contrast, pursuant to the present invention, a quickconnect is not used to connect to the cold plate, though quick connectsmay be employed for the connections to each individual blade.

Koltuniak, et al., U.S. Pat. No. 3,749,981 describes a modular powersupply wherein the power modules, each with its own fans, are mountedinside a sealed cabinet. Also mounted inside the cabinet are coolingmodules, each with its own fan and heat exchanger. This patentrepresents an early example in the technology of an air-recirculationsystem requiring shared airflow plenums that occupy valuable space.

Ward, et al., U.S. Pat. No. 3,387,648 pertains to a cabinet-enclosedcooling system for electronic modules mounted on a modular chassis,wherein the chassis is extensible from the cabinet. This is an exampleof an air-recirculation system that requires, at the front and back ofthe assembly, shared vertical air plenums which unnecessarily occupyvaluable space.

In implementing the construction of high-performance computer systems,it is desirable to be able to electrically interconnect as manyprocessor chips and memory cards as possible while using conventionaland economically priced electronic packaging methods. Thereby, the moredensely and closely packed the electronics are, the more difficult theyare to cool, because space is required for air circulation and for heatsinks. One method of achieving dense packaging of the electroniccomponents is to build modular units called “blades”, each of whichcontains one or more processors and memory card(s). Multiple blades arethen plugged into a common electrical backplane, or midplane, which,because of its high wiring density, provides for a high-speed andcost-effective inter-blade communication. Moreover, the modularity ofblades allows for the sharing of common system resources, andfacilitates servicing and configuration changes. Blade-type packaging isnot limited to computer systems, but may also be employed for switchsystems, or other types of information processing, and for matchingand/or mixing of different functions within a single rack or enclosure.

Two features of conventional blade-style packaging essentially limit theperformance achievable by the electronic components located within arack:

1. Front-To-Back Airflow

Racks with blade-style packaging frequently employ vertical backplanes(or midplanes) in conjunction with front-to-back airflow coolingarrangements, thereby requiring airflow holes to be formed in thebackplane. Such holes, to a significant extent, block wiring channels inthe backplane, thereby greatly reducing the number of I/O's(input/output electrical signaling interconnections) available forconnection to the attached blades. Moreover, in such a rack, therelatively small airflow cross-section provided by the holes in thebackplane limits total power dissipation to about 30 kW. This aspect isdisclosed in the publication by M. J. Crippen, et al., “BladeCenterpackaging, power, and cooling”, IBM J. Res. & Dev., Vol. 49 No. 6,November 2005, pp. 887-904.

2. Total Reliance on Air-Cooling

As a cooling fluid, air is advantageous vis-à-vis water because iteffortlessly bathes myriad heat-producing electronic devices in a safe,insulating cooling fluid. However, air is disadvantageous in comparisonwith water because its small heat capacity per unit volume, 3500 timessmaller than water, limits the power density that may be cooled, andrequires a considerable amount of airflow space, which restrictspackaging density.

The above-mentioned features of conventional blade-style packaging,front-to-back airflow and total reliance on air cooling, must be clearlyimproved upon in order to solve the following problems, which arecurrently in evidence:

(a) limited total power that can be dissipated in a blade-style rack,

(b) limited packaging density due to space required for airflow,

(b) high engineering cost of customized airflow solutions forconventional raised-floor data centers,

(c) excessive data-center noise encountered due to air movers andairflow, and

(d) discomfort encountered by personnel in data centers due tonon-uniform air temperatures.

SUMMARY OF THE INVENTION

The current invention implements two ideas in a unique and novel manner:first, the use of vertical airflow with vertical backplanes ormidplanes; and second, the combined use of both air and water ascoolants, in an arrangement that exploits the strengths of both fluids.

As a result, whereas conventional, air-cooled, blade-style packaginglimits total power to 30 kW in a noisy rack occupying 2′×3′ of floorspace (5 kW/ft²), a prototype embodiment of the present invention, witha realistic mix of heat-producing components, will successfully cool atotal power of 81 kW in a quiet rack occupying 2.7′×5′ of floor space (6kW/ft²). Moreover, the prototype embodiment indicates that futureembodiments could readily cool over 100 kW in such a rack (>7.4 kW/ft²).

Accordingly, the present invention provides for a method and anapparatus for cooling, preferably within an enclosure, a diversity ofheat-generating components, with at least some of the components havinghigh power densities and others having low power densities. Directliquid cooling is used to remove heat generated by a relatively smallnumber of high-power-density components exemplified by microprocessorchips, whereas novel, closed-loop, liquid-assisted air cooling is usedto remove heat generated by a relatively large number oflow-watt-density components exemplified by memory chips.

In effectuating direct liquid cooling, microchannel coolers or othertypes of cold heads are attached directly to the high-power-densitycomponents. In effectuating closed-loop liquid-assisted air cooling, airtravels upwardly in the front half of an enclosure through relativelynarrow rectangular packages, referred to as “blades”, which containdiverse heat-generating components, and which are positioned in multiplerows located one above the other. Air-to-liquid heat exchangers areinterleaved between rows of blades in order to cool the air emergingfrom each respective blade row before entering the next row. The heatthat the air removes from the blade row is transferred in its entiretyto the liquid, and is thereby removed from the enclosure, with the airbeing thereby assisted in its cooling task by means of the liquid. Theblades are ordinarily attached to the front side of one or more central,vertical circuit cards, referred to as “midplanes”. At the top of thefront stack of blades, the air then travels through a first set of airmovers that divert the air towards the rear half of the enclosure, andinto a first high-pressure plenum. From this top-and-rear-locatedhigh-pressure plenum, the air then travels downwardly within the rearhalf of the enclosure through additional rows of blades attached to theother side of the midplanes, and finally through a second set of airmovers that divert the air towards the front half of the enclosure andinto a second high-pressure plenum. From this bottom-and-front-locatedhigh-pressure plenum, the air again travels upwardly through the frontblades, thereby completing a closed loop. This closed-loop,liquid-assisted air cooling architecture enables multiple blades to beconnected to the front and rear of the midplanes, thereby facilitatingthe provision of low-cost, densely arranged, high-performance electricalinterconnections within the rack. Because air flowing through the bladestravels substantially in parallel with the respective midplane, themidplane does not need to be provided with air-circulation holes, whichwould tend to block wiring channels, thereby imparting an importantadvantage to this structural arrangement. Moreover, no verticallydirected air plenums, which occupy valuable floor space, are needed inthis structure.

On each side of the midplanes, each horizontal row of blades ismechanically supported by a blade cage having bottom and top surfaces,which are substantially open in order to allow for a large volume of avertical flow of air at a low pressure loss, another important advantageof this arrangement in comparison with the conventional practice offlowing air through small holes formed in the midplanes or backplanes.

Because the air-to-liquid heat exchanger interposed downstream of eachblade cage removes from the air, on an average, all heat absorbedtherein, the combination of a blade cage and a heat exchanger isthermally neutral for the air; in essence, the air temperature increasesfrom T₁ to T₂ as it passes through a blade cage, but then decreases fromT₂ to T₁ as it passes through the heat exchanger immediately downstreamthereof. The air thereby traverses its closed loop, through M bladecages and M heat exchangers, without any net increase in itstemperature. Inasmuch as the air loop is enclosed in the rack, and thewalls of the rack are insulated with an acoustic-transmission-lossmaterial, the invention provides for a much quieter, more comfortableroom for personnel than that encountered in conventional installationswhere noisy air movers located in the rack expel to the room largeamounts of hot air, which must be collected by air-conditioning unitsthat create additional noise. Eliminating this prior-art construction isyet another important advantage of the present invention.

The liquid in the air-to-liquid heat exchangers is normally carried inpiping that is distinct from the piping used to carry the liquid fordirect-liquid cooling, so the two liquids may differ, but are typicallyboth water, often with anti-corrosion, algicide, and other additives. Inorder to save vertical space, a coolant-distribution manifolds fordirect-liquid cooling of each blade cage is placed immediately in frontof a heat exchanger adjacent to the blade cage, thereby ensuring that apair of hoses, which connect each blade to the inlet or outletmanifolds, may be disconnected for blade removal without having todisturb other blades. Furthermore, in order to save space in thedirection normal to the midplanes, the quick disconnects for themanifolds are mounted at an angle. Such efficient packaging permits alarge quantity of electronics to be housed within a small amount ofspace, thereby presenting another advantage of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a perspective representation of a preferredembodiment of a computer enclosure of the invention showing majorinterior components thereof;

FIGS. 2A and 2B illustrate schematically, respectively front and sideviews of the embodiment of the enclosure of FIG. 1 and its majorinterior components;

FIG. 3 illustrates a perspective view of the enclosure of FIG. 1 showingthe exterior doors thereof;

FIG. 4 illustrates a perspective view of a thermally neutral front unit,which is fully equipped with blades;

FIG. 5 illustrates a perspective view of the thermally neutral frontunit of FIG. 4, shown partially disassembled;

FIG. 6 illustrates a perspective view of a blade cage for the unit ofFIG. 4;

FIG. 7 illustrates a perspective view of a heat exchanger andquick-connect manifold;

FIGS. 8A and 8B illustrate, respectively schematic front and side viewsshowing the enclosure-level plumbing;

FIG. 9 illustrates a perspective view of a thermally neutral front unitshowing the enclosure-level plumbing of FIGS. 5A and 8B;

FIG. 10 illustrates a perspective view of a left-side view of aprototype blade;

FIG. 11 illustrates a perspective view of a right-side view of theprototype blade;

FIG. 12 illustrates a perspective view of a top view of the prototypeblade;

FIG. 13 illustrates a perspective view of a bottom and rear view of afan, shown with a box;

FIG. 14 illustrates a perspective view of four fans, shown withoutboxes;

FIG. 15 illustrates a perspective view of the four fans, shown withboxes;

FIG. 16 illustrates a perspective view of a front view of an air-movingassembly;

FIG. 17 illustrates a perspective view of a bottom view of theair-moving assembly;

FIG. 18 illustrates a perspective view showing the detachment of a fanfrom the air-moving assembly;

FIG. 19 illustrates a perspective view of a lower air-moving assemblyand plenum box;

FIG. 20 illustrates a perspective view of an upper air-moving assemblyshowing an arrangement that mitigates the detrimental thermal effect ofa fan failure.

FIG. 21 illustrates a perspective view of an upper air-moving assembly,showing an alternative arrangement that mitigates the detrimentalthermal effect of a fan failure.

FIG. 22 illustrates a perspective view of the main computer enclosurewith an attached enclosure providing for bulk power supplies.

BRIEF DESCRIPTION OF THE INVENTION

Referring now specifically to the disclosure, FIG. 1 shows a perspectiverepresentation, and FIGS. 2A and 2B show, respectively equivalent frontand side elevations, of a preferred embodiment of an enclosure 1, suchas for computer or electronic components, and its heat-producingcontents. External walls and doors 2, which cover the frame 3 ofenclosure 1, shown in FIG. 3, are omitted in FIG. 1 for purposes ofvisual accessibility to the interior. FIGS. 2A and 2B show an imaginary,Cartesian xyz coordinate system 4 whose xz plane divides the enclosure 1into a front region 5 (y<0) and a rear region 6 (y>0), and whose yzplane divides the enclosure into two halves (x>0 and x<0). All otherFigures show similar xyz coordinate systems whose axes extend in allcases, in parallel with the yz coordinate system 4 of FIGS. 2A and 2B,but the origins of which may differ.

Many of the drawing figures are true-scale representative diminutions ofa full-scale, thermal-prototype enclosure built to embody the conceptscontained in this application. Frequent reference will be madehereinbelow to the specifics of this prototype embodiment, but it shouldbe understood that the invention is not limited thereto. The x×y×zdimensions of the thermal-prototype enclosure shown in FIG. 3, includingthe external walls and doors 2, but excluding bulk power supplies thatare housed in a separate enclosure (as shown in FIG. 22), are0.81×1.52×2.13 meters (32×60×84 inches). Experiments indicate that byusing the cooling schemes described herein, up to 81 kW of heat (55.8 kWliquid-assisted air cooled, 25.5 kW direct-liquid cooled) may bedissipated in the thermal-prototype enclosure 1 with only modestcomponent temperatures (71° C. worst case, based on over 11,000 measuredlocations).

Within the front region 5, a central-front region {y<0, z₁<z<z₂}encloses an integer number M_(F) of front heat exchangers 7, M_(F) frontblade rows 8, and M_(F) front quick-connect manifolds 9. Along the zdirection, the front blade rows 8 are interleaved with the front heatexchangers 7. A front quick-connect manifold 9, which supplies the bladerow 8 immediately thereabove with cooling liquid for direct-liquidcooling, is located on the −y side of each front heat exchanger 7. Eachfront blade row 8 comprises an integer number N_(F) of front blades 10,which are arrayed along the x direction. Each front blade 10 is apackage, generally having the shape of a rectangular parallelepiped,that contains front heat-producing components 11. The heat generated bythe heat-producing components 11 is generally the result of Jouleheating encountered in electrical circuits.

Similarly, within the rear region 6, a central-rear region {y>0,z₁<z<z₂} encloses an integer number M_(R) of rear heat exchangers 12,M_(R) rear blade rows 13, and M_(R) rear quick-connect manifolds 14.Along the z direction, the rear blade rows 13 are interleaved with therear heat exchangers 12. The interleaved ordering of rear blade rows 13and rear heat exchangers 12 is opposite to that of front blade rows 8and front heat exchangers 7; that is, where the z-wise order of frontheat exchangers 7 and front blade rows 8 (from bottom to top) is7-8-7-8-7-8-7-8 as shown in the drawings, then the z-wise order of rearheat exchangers 12 and rear blade rows 13 (from bottom to top) is13-12-13-12-13-12-13-12. One of the rear quick-connect manifolds 14 islocated on the +y side of each rear heat exchanger 12. Each rear bladerow 13 comprises an integer number N_(R) of rear blades 15 that containrear heat-producing components 16.

Although front blade rows 8 are illustrated as being identical to rearblade rows 13, and front heat exchangers 7 are illustrated as beingidentical to rear heat exchangers 12, it is possible to accommodatedissimilar blade rows and heat exchangers, provided that the heatexchanger immediately downstream of each blade row is sizedappropriately to remove the heat load thereof. Moreover, althoughM_(F)=N_(F)=4 is shown, other values of M_(F) and N_(F), which may notbe necessarily equal, are within the scope of the disclosure. Likewise,although M_(R)=N_(R)=4 is shown, other values of M_(R) and N_(R), notnecessarily equal, are also applicable. Furthermore, although the z-wiseorders 7-8-7-8-7-8-7-8 and 13-12-13-12-13-12-13-12 are shown in thefront region 5 and rear region 6 respectively, the opposite orders,8-7-8-7-8-7-8-7 and 12-13-12-13-12-13-12-13, are also possible. Inaddition, z-wise arrangements using fewer heat exchangers, such as8-8-7-8-8-7 and 12-13-13-12-13-13, are also contemplated.

Each front blade 10 and each rear blade 15 is electrically connected toa midplane (17, 18), which is an electrical circuit card upstanding inthe xz plane, and whose function resides in providing electrical powerto, and electrical communication between, the blades that are connectedthereto. The enclosure 1 may contain one or more midplanes, although onelarge midplane is often preferred so as to provide connectivity betweenas many blades as possible. However, because the maximum size of circuitcards may be limited by logistics of manufacturability, two or moremidplanes may exist in the enclosure 1. As an example, FIG. 2B shows twomidplanes, a lower midplane 17 and an upper midplane 18. In this case,with four rows of front blades (M_(F)=4) and four rows of rear blades(M_(R)=4), all front blades 10 in the lower two front blade rows 8connect to the −y surface of the lower midplane 17 via front midplaneconnectors 19, and all rear blades 15 in the lower two rear blade rows13 connect to the +y surface of the lower midplane 17 via rear midplaneconnectors 20. Likewise, all front blades 10 in the upper two frontblade rows 8 connect to the −y surface of the upper midplane 18 viafront midplane connector 19, and all rear blades 15 in the upper tworear blade rows 13 connect to the +y surface of the upper midplane 18via rear midplane connectors 20 where at least one connector is providedfor each blade, but multiple connectors could also be used for eachblade.

In general, although not necessarily, the front heat-producingcomponents 11 may be divided into two types: low-power-density frontheat-producing components 21 and high-power-density front heat-producingcomponents 22. Similarly, the rear heat-producing components 16 may bedivided into low-power-density rear heat-producing components 27 andhigh-power-density rear heat-producing components 28. Alow-power-density heat-producing component may be defined, for example,as having a worst-case surface heat flux less than P; ahigh-power-density heat-producing component may then be defined ashaving a worst-case surface heat flux that exceeds P, where as typicalvalue of P may be 75 W/cm². Although, in FIG. 2B, high-power-densitycomponents 22, 28 are shown only at the centers of the blades in the ydirection, the invention is not restricted thereto; in general,low-power-density components 21, 27 and high-power-density components22, 28 may be located anywhere within the three-dimensional volumeoccupied by front blades 10 or rear blades 15. Furthermore,classification of a component as “low-power-density” or“high-power-density” depends largely on the distribution of heatgeneration therewithin, because not only the peak heat flux P at a“hotspot”, but also the physical size of the hotspot, must beconsidered. Moreover, P also depends on the highest permissibletemperature T_(max) of a component: the lower the required value ofT_(max), the lower the definition of P must be.

The represented prototype embodiment uses mockup heat-producingcomponents, such as resistors and thermal test chips, instead of realheat-producing components such as processor and memory chips. In orderto monitor the temperature of the mockup heat-producing components, over11,000 temperature sensors are placed near selected componentsthroughout the prototype enclosure 1. Power dissipation of the mockuplow-power-density heat-producing components is 1744 watts per blade;power dissipation of the mockup high-power-density components is 800watts per blade. The prototype enclosure has space for 32 blades, asshown in FIGS. 2A and 2B, and thus can accommodate a total enclosurepower of 81 kW, which far exceeds the capabilities of conventionalequipment enclosures. The plan-form power density of the prototypeembodiment is 81 kW/13.3 ft²

6 kW/ft². This far exceeds the power density of typical data centers,which are designed to be cooled by the use of circulating air.

For the two types of heat-producing components, the present inventionprovides two different cooling solutions; namely, closed-loopliquid-assisted air cooling for low-power-density components 21, 27, anddirect-liquid cooling for high-power-density components 22, 28,

Referring to FIG. 1, closed-loop liquid-assisted air-cooling employs aclosed loop 23 of circulating air that is confined within the enclosure1. The closed loop 23 comprises a front airflow segment 24 flowingtowards +z in the central-front region (y<0; z₁<z<z₂) of the enclosure1, a top airflow segment 25 flowing towards +y in a top region {z>z₂} ofthe enclosure, a rear airflow segment 26 flowing towards −z in thecentral-rear region (y>0; z₁<z<z₂) of the enclosure, and a bottomairflow segment 27 flowing towards −y in a bottom region {z<z₁} of theenclosure 1.

As air flows along the front airflow segment 24, it is alternatelycooled by one of the front heat exchangers 7 and then heated by one ofthe front blade rows 8; this cooling-and-heating cycle occurs M_(F)times along the front airflow segment 24. Similarly, as air flows alongthe rear airflow segment 26, it is alternately heated by one of the rearblade rows 13 and then cooled by one of the rear heat exchangers 12;with this heating-and-cooling cycle occurring M_(R) times along the rearairflow segment 26. Thus, along the front airflow segment 24, eachadjacent combination of front heat exchanger 7, front blade row 8, andfront quick-connect manifold 9 represents a thermally neutral front unit28; whereby on average, each streamline of air in the front airflowsegment is cooled by one of the heat exchangers from a temperature T₂ toa lower temperature T₁, but is then reheated by the following frontblade row from temperature T₁ to the original temperature T₂. Likewise,each combination of rear heat exchanger 12, rear blade row 13, and rearquick-connect manifold 14 represents a thermally neutral rear unit 29.Because the aforesaid z-wise order 7-8-7-8-7-8-7 of front heatexchangers 7 and front blade rows 8 is arranged opposite to the z-wiseorder 13-12-13-12-13-12-13-12 of rear heat exchangers 12 and rear bladerows 13, the desired, alternating order of heat exchangers and bladerows is maintained, in the air-stream direction, as the air moves (atthe top of the enclosure 1) from the front region 5 to the rear region6, and conversely as it moves (at the bottom of the enclosure 1) fromback to front.

In FIG. 1, this closed loop 23 is depicted diagrammatically as therectangle that is delineated by an upper-front airflow corner 30, anupper-rear airflow corner 31, a lower-rear airflow corner 32, and alower-front airflow corner 33. Movement of air along the closed air loop23 is driven by an upper air-moving assembly 34 (FIGS. 2A and 2B) thatcomprises an integral number K_(U) of upper fans, such as 35, 36, 37, 38located in a top-front region {y<0; z>z₂} of the enclosure 1, as well asby a lower air-moving assembly 39 that comprises an integral numberK_(L) of lower fans such as 40, 41, 42, 43, located in a bottom-rearregion {y>0; z<z₁} of the enclosure 1. In the prototype embodiment shownin FIG. 1 and FIGS. 2A and 2B, the number of upper fans in the firstair-moving assembly 34 is K_(U)=4; likewise, the number of lower fans inthe second air-moving assembly 39 is K_(L)=4. In such an embodiment, theclosed air path 23, shown schematically in FIG. 1 as the singlerectangle (30, 31, 32, 33), is more accurately represented, as shown inFIGS. 2A and 2B, as two concentric rectangles; namely, an innerrectangle 44 and an outer rectangle 45. The inner rectangle 44represents air driven by the pair of top-inner fans (37,38) and the pairof bottom-inner fans (42, 43). The outer rectangle 45 represents airdriven by the pair of upper-outer fans (35, 36) and the pair oflower-outer fans (40, 41). It should be emphasized that, notwithstandingthis illustration of the airflow as one discrete loop 23 or as twodiscrete loops 44 and 45, in actuality an infinite number of parallelstreamlines flow along such closed paths, bathing the entire volumeoccupied by the blade rows (8, 13), the heat exchangers (7, 12), and theair-moving assemblies (34, 39) in the air stream.

Other arrangements of air movers are also within the scope of theinvention. For example, arrays of axial-flow fans may be interleavedbetween blade cages and heat exchangers so as to replace or supplementthe air-moving power of the shown centrifugal fans.

In the closed loop 23, the only empty spaces needed for air plenums area top-rear region 46 {y>0; z<Z₂} and a bottom-front region 47 {y>0;<Z₂}. Consequently, no floor space is lost along the sides or front andback of the rack for air distribution, because there are no air plenumsthat extend vertically in the enclosure. Thus, except for the spaceoccupied by the external walls or doors 2, the full “foot-print” of therack is available for electronics, which are contained in the blades(10, 15).

Air in the closed loop 23 increases in temperature from a cooltemperature T₁ to a warm temperature T₂ as it flows through each bladerow (8. 13), because the air convectively absorbs heat dissipated by thelow-power-density components 21, 27 therein. However, the heated air isimmediately restored to the cool temperature T₁ as it flows through theheat exchanger (7, 12) that immediately follows the blade row in theair-stream direction Thus, each adjacent combination of front heatexchanger 7 and front blade row 8 is thermally neutral for the air.Consequently, in traversing through the closed loop 23, the air isheated and cooled M_(F)+M_(R) times, with no net change in temperaturebeing encountered during steady-state operation.

The heat exchangers 7 and 12 are air-to-liquid heat exchangers in whoseliquid side is circulated an air-assisting liquid 48. All heatdissipated by the low-power-density components is removed from theenclosure 1 by the air-assisting liquid 48, which is typically, but notnecessarily, water that communicates with an external chilled-watersystem (not shown). This chilled-water system must provide, by meanswell known in the art, reasonably-clean, non-corrosive, above-dew-pointwater for use in the heat exchangers (7, 12). Clean water is required toprevent heat-exchanger fouling (which can compromise heat-transferperformance); non-corrosive water is required to prevent corrosion ofmetal plumbing; and above-dew-point water is desired to avoid water inthe air from condensing on the surfaces of the heat-exchangers.

For direct-liquid cooling of the high-power-density heat-producingcomponents 22, 28, a direct-cooling liquid 49 is conveyed thereto in amanner described below, whence the components' heat load is primarilytransferred directly to the direct-cooling liquid 49 by solid-to-solidconduction and solid-to-liquid convection. Thus, virtually all heatdissipated by the high-power-density components is removed from theenclosure 1 by the direct-cooling liquid 49, which is typically watercommunicating with an external chilled-water system. Again, thechilled-water system must provide, by means well known in the art,filtered, non-corrosive, above-dew-point water for use in direct liquidcooling of the high power density components 22, 28.

Because both types of cooling, i.e., closed-loop liquid-assisted aircooling and direct-liquid cooling, reject heat to liquids within theenclosure 1, the entire enclosure appears to an outside observer to beliquid cooled. Yet in reality, air cooling is used to advantageinternally, because the low-power-density components 21, 27, treatedwith liquid-assisted air cooling, are ordinarily numerous and thereforedifficult and expensive to treat with direct-liquid cooling.

In principle, the direct-cooling liquid 49 and the air-assisting liquid48 may be independent of each other, and may resultingly operate atdifferent temperatures, thereby allowing, for example, verylow-temperature high-power-density components (e.g. processors) to becombined with higher-temperature, air-cooled low-power-densitycomponents (e.g. memory chips). It is not ordinarily contemplated tohave any components at temperature low enough so that condensation formsunder typical computer machine room conditions. However, if very orextremely low temperatures are required, the relatively well-sealedvolume of air inside the enclosure 1 may be dehumidified by suitablemeans well known in the art.

Discussed hereinbelow in specific detail are various components andoperating aspects of the inventive apparatus employed for implementationof the novel cooling method.

Thermally Neutral Units

The structure of a thermally neutral front unit 28, which comprises oneof the front heat exchangers 7, the front blade row 8 directlythereabove, and the front quick-connect manifold 9 directly in frontthereof, is described in detail below. The structure of a rear thermallyneutral unit, which comprises one of the rear heat exchanger 12, therear blade row 13 directly therebeneath, and the rear quick-connectmanifold 14 directly therebehind, is similar, possibly even identical,except for the inverted z-wise order of components, as discussedpreviously in connection with FIGS. 2A and 2B.

FIGS. 4 and 5 illustrate two views of one of the M_(F) thermally neutralfront units 2B, which, for explanatory purposes, is assumed to be thethermally neutral unit at the lower left of FIG. 2B. The front blade row8 comprises N_(F) front blades 10 housed together in a front blade cage50, which mechanically supports the front blades within the enclosure 1,and which locates the blades 10 for slidable connection in the −ydirection relative to the midplane 17, to which blades in a rear bladecage (not shown in FIGS. 4 and 5) may also be connected. As shown, themidplane preferably extends in the z direction above this thermallyneutral front unit (assumed above to be the lowest in the enclosure 1),so that instead of the midplane being shared only by the lowesthorizontal blade rows front and rear, it is shared by more than onehorizontal row. For example, FIG. 2 shows lower and upper midplanes (17,18) that each provide connectivity between four blade rows: two frontblade rows 8 and two rear blade rows 13. The midplane 17 may also extendbeyond the blade cage 50 in the !x direction, as shown, to allow spacefor connections that bring electrical power to the midplane, which inturn distributes power to the front and rear blade rows (8, 13).

In a prototype embodiment, each prototype front blade 10 has dimensionsof 120 mm×560 mm×305 mm in the x, y, and z directions, respectively, andeach prototype blade cage 50 has xyz dimensions of 573 mm×605 mm×311 mm.Each prototype heat exchanger has dimensions of 540 mm×605 mm×48 mm inthe x, y, and z directions, respectively. The prototype thermallyneutral units 28 are stacked on a 375-mm pitch in the z direction.

FIG. 4 shows the N_(F)=4 front blades located in the positions theywould normally occupy during operation when plugged at right angles intothe midplane 17. FIG. 5 shows the leftmost blade disconnected from themidplane and partially withdrawn from the blade cage, therebyillustrating the manner in which a blade is removed from the blade cagefor servicing or replacement. FIG. 5 also shows the rightmost two bladesomitted, thereby revealing the front air-to-liquid heat exchanger 7therebeneath.

Arrows in FIGS. 4 and 5 indicate the flow of cooling fluids through thethermally neutral front unit 28. The closed loop 23 of circulating air,which cools the low-power-density components 21, flows in the +zdirection. In the prototype embodiment, the volumetric flow rate of airalong the closed loop 23 is determined by a detailed scan of measuredvelocities over a typical blade. The scanned velocities vary betweenabout 3.0 and 5.0 m/s. The average air velocity is 3.4 m/s over the 480mm×535 mm cross-sectional area of the loop. Integrating the velocityover the cross-sectional area, the total volumetric flow rate along theclosed loop 23 is 0.873 m³/s (1850 standard cubic feet per minute).Because the air is cooled by heat exchange to liquid eight times aroundthe prototype-embodiment's closed loop 23, it is to be appreciated thatthis 1850 CFM is equivalent to 14,800 CFM of conventional air cooling,because the latter does not use multiple heat exchanges from air toliquid. Such a large equivalent flow rate of air is extremely difficultto accomplish (with an enclosure of the size used in the prototypeembodiment) by means other than those taught by the present invention.

The air-assisting liquid 48, which cools the closed loop 23 ofcirculating air before it enters the next blade-row thereabove, entersthe heat exchanger 7 through a heat-exchanger supply fitting 51 andexits through a heat-exchanger return fitting 52. In a prototypeembodiment, the air-assisting liquid employed is water, with avolumetric flow rate through each heat exchanger of approximately 11.4liter/minute (3.0 gallons/minute).

Direct-cooling liquid 49, which cools high-power-density components 22,enters the quick-connect manifold 9 through a manifold supply pipe 53and exits through a manifold return pipe 54. In the prototypeembodiment, the direct-cooling liquid is preferably water, with avolumetric flow rate to the quick-connect manifold 9 of approximately8.0 liters/minute (2.11 gallons/minute), which imparts a flow to eachidentical blade of 2.0 liters/minute (0.53 gallons/minute).

Blade Cages

FIG. 6 illustrates the front blade cage 50, showing all blades havingbeen removed, thereby revealing its structure as possessing solid sidesurfaces 55; an open front cage surface 56 to allow insertion of frontblades; a slotted rear cage surface 57 with slots 58 to allow connectorsnear the +y edge of the front blades to mate to the midplane connectors19 on the front surface of one of the midplanes (17, 18); a bottom cagesurface 59 having large rectangular bottom-flow-through holes 60; and atop cage surface 61 having similar, large rectangular top-flow-throughholes 62. The bottom flow-through holes 60 and top-flow-through holes 62together allow for airflow through the front blades in the +z direction.Between the N_(F) bottom-flow-through holes are (N_(F)−1) bottom bladeguides 63 formed from the sheet metal of the bottom cage surface 59.Likewise, between the top-flow-through holes, top blade guides 64 areformed from the sheet metal of the top cage surface 61. Each blade guide(63, 64) has a U-shaped cross section in the xz plane that providesguidance for the blades as they prepare to engage the midplaneconnectors 19. The U-shaped cross section also imparts considerablestiffness to the guide itself, thereby preventing excessive bending ofthe bottom guides 63 under the weight of the front blades 10, which bearon the bottom guides as the blades are inserted and removed.

As shown in FIG. 5, each front blade 10 has a left sheet-metal skin 65on its −x face and a right sheet-metal skin 66 on its +x face. Each ofthese faces has a hemmed top edge 67 that slides within one of the topblade guides 64 while the blade is being inserted or withdrawn, and ahemmed bottom edge 68 that similarly slides within one of the bottomblade guides 63. Hemming the edges prevents galling the blade guides(63, 64) as the edges slide on them. Each bottom blade guide 63 is wideenough to accept two hemmed bottom edges 68; one belonging to the rightsheet-metal skin 66 of the blade to its left, and the other belonging tothe left sheet-metal skin 65 of the blade to its right. Likewise, eachtop blade guide 64 is wide enough to accept two hemmed top edges 67, onefrom a blade to its left, and another from a blade to its right. Asshown in FIG. 6, special guides 69 at the extreme left and right of theblade cage, both top and bottom, are wide enough to accept one hemmededge only. To assist in aligning the blade in the x direction so thatthe hemmed edges properly engage the blade guides, tapered startingblocks 70, affixed to the bottom and top cage surfaces (59, 61) areprovided between blade guides (63, 64, 69).

Quick-Connect Manifolds

Referring to FIG. 4, the direct-cooling liquid 49 in the manifold supplypipe 53 is supplied to one of the blades 10, for example the secondblade from the right in FIG. 4, by flowing first through a supply quickconnect 71 that is attached (for the purpose of minimizing the ydimension of the enclosure 1) at an acute angle to the manifold supplypipe 53, then through a manifold-supply elbow fitting 72, thereafterthrough a flexible supply hose 73, and finally through a blade-supplyelbow fitting 74. If it is desired to balance flow between variousblades, a control valve may be inserted between each flexible supplyhose 73 and the corresponding blade-supply elbow fitting 74. In aprototype embodiment, Swagelok model SS-1RS6 needle valves may be usedfor this purpose. Flow of the direct-cooling liquid 49 through the bladeitself is described hereinbelow. The direct-cooling liquid is returnedfrom the blade to the manifold return pipe 54 by flowing first through ablade-return elbow fitting 75, then through a flexible return hose 76,then through a manifold-return elbow fitting 77, and finally through areturn quick connect 78. The flexible supply hose 73 and flexible returnhose 76 must be flexible to permit operation of the supply quick connect71 and the return quick connect 78. For example, 85 durometerpolyurethane hose may be suitable for hoses 73, 76. In the prototypeembodiment, 6.35-mm-I.D., 9.53-mm-O.D. hose of this type is used, withSwagelok connections.

A quick-connect, well known in the art, is a two-piece plumbingconnection that provides rapid, easy, virtually dripless connection anddisconnection of a fluid line. The two pieces are referred to as “body”and “stem”; the body is the larger (female) half of the connection; thestem is the smaller (male) half. Each piece has a shut-off valve,whereby when the two halves are disconnected with fluid flowing in theline, the flow is automatically stopped in both disconnected halves ofthe line. When the two halves are re-connected, the flow automaticallyrestarts. Such convenient disconnection and reconnection are essentialto the equipment in this invention, inasmuch as the equipment is proneto occasional failure, and thus requires occasional servicing orreplacement. Although high-quality quick connects are quite reliable andvirtually dripless, the supply and return quick connects 71, 78 in thepreferred embodiment are located, as shown in FIG. 4, in front of theblades, with no electronic components located directly therebeneath.Such a location is preferred to avoid any possibility of direct-coolingliquid 49 dripping onto the electronics.

As shown by the empty blade positions in FIG. 5, the manifold supplypipe has attached thereto, at N_(F) locations (one for each of the N_(F)blades in the blade row), an angle block 79 and a quick-connect supplybody 80. The angle block is designed to orient the quick connects 71, 78at an acute angle to the manifold pipes 53, 54, rather than at a rightangle, in order to minimize the y dimension of the enclosure 1.Similarly, the manifold return pipe 54 has attached thereto, at N_(F)locations, one of the angle blocks 79 and a quick-connect return stem81. Each blade 10 has a quick-connect supply stem 82 (seen more clearlyin FIG. 10) that leads to the supply hose 73 and a quick-connect returnbody 83 that leads to the return hose 76. One of the blades 10 may bequickly connected to the flow of direct-cooling liquid 49 by connectingthe blade's quick-connect supply stem 82 to the manifold's quick-connectsupply body 80, and by also connecting the blade's quick-connect returnbody 83 to the manifold's quick-connect return stem 81. The connectionsare arranged this way, with supply and return connections havingopposite genders, to avoid any possibility of erroneous connection. In aprototype embodiment, the quick-connect bodies and stems may be SwagelokSS-QTM2A-B-4PM and QTM2-D-4PM, respectively.

Heat Exchangers

In a prototype embodiment, the front heat exchangers 7 and rear heatexchangers 12 are identical; details of such a heat exchanger, as wellas the quick-connect manifold 9 attached thereto, are shown in FIG. 7.The construction of this device, known as a copper-tube, aluminum-finair-to-liquid heat exchanger, is well known in the art of heat-exchangerfabrication. It comprises a copper supply fitting 84 that supplies theair-assisting liquid 48 from an external chilled-liquid system (notshown) to a copper-pipe supply header 85, and a copper return fitting 86that returns the air-assisting liquid 48 from a copper-pipe returnheader 87 to the chilled-liquid system. In the heat exchanger, flow ofthe air-assisting liquid occurs through an integer number N_(C) ofcopper piping circuits 88 that in parallel convey the air-assistingliquid from the supply header 85 to the return header 87. One end ofeach piping circuit 88 is connected to the supply header 85 by a supplyfeeder 89; the other end of each piping circuit is connected to thereturn header 87 by a return feeder 90. Each piping circuit 88 comprisesan integer number N_(P) of straight copper pipes 91 that extend back andforth along the +x and −x directions through tight holes in finelyspaced aluminum fins 92. The N_(P) straight copper pipes are connectedat their ends by N_(P)−1 U-turn copper fittings 93, thereby to form acontinuous meandering path from supply header to return header. Allalong this meandering path, the air-assisting liquid 48 absorbs heat;heat is transferred first by convection from hot air in the closed airloop 23 to the aluminum fins 92 and straight copper pipes 91, then byconduction through the aluminum fins 92 and the straight copper pipes91, and finally by convection from the interior of the copper pipes tothe air-assisting liquid 48 within the tubes. Surrounding the finnedarea of the heat exchanger 7 is a four-sided C-channel frame 94, 95, 96,97 whose right side 95 is shown partially cutaway in FIG. 7 in order toreveal the piping circuits 88. Holes in the frame sides 95, 97 supportthe piping circuits.

In the prototype embodiment, the number of piping circuits 88 isN_(C)=7, the number of passes per circuit is N_(P)=6, the outer diameterof copper pipes in piping circuits 88 is 9.5 mm, the outer diameter ofthe header pipes (85, 87) is 16 mm, the fins 92 are 0.1 mm thick on 1.5mm centers, and the height of each fin in the z direction is 44 mm. Thefinned area of the heat exchanger covers the full cross-sectional areaof the front blade row 8 that it must cool; for the prototypeembodiment, the x and y dimensions of this area are 480 mm and 530 mm,respectively.

A space-saving advantage of the prototype embodiment resides in that thequick-connect manifold 9 nestles inside the C-channel-frame's frontmember 94, being attached thereto by means of scalloped clamps 98 thatcradle the manifold supply and return pipes 53, 54. The front half ofeach clamp, visible in FIG. 7, cradles the front surfaces of the pipes53, 54; while the rear half of each clamp, not visible in FIG. 7,cradles the rear surfaces of the pipes. The rear half of each clamp isaffixed to the C-channel-frame's front member 94. To secure the pipes53, 54 to the front member 94, the front and rear halves of each clamp98 are pulled together by a screw that passes through a hole 99 in thefront half of the scalloped clamp, passes between the two pipes 53, 54,and engages a threaded hole in the rear half of the scalloped clamp.

Plumbing Connections for Heat Exchangers

The heat-exchanger's supply header 85 is connected to a heat-exchangersupply riser 100, shown schematically in FIG. 8 and pictorially in FIG.9, that supplies the air-assisting liquid 48 from a first chilled-liquidsystem (not shown) at a supply temperature T_(S1) to an entire column ofheat exchangers, which are connected in parallel. If the firstchilled-liquid system is a chilled-water system, as is well known in theart, then the temperature of water in the heat-exchanger supply riser100 is typically T_(S1)=18 to 20° C. in order to be safely above thedew-point temperature of typical computer-room environments.

The heat exchanger's return header 87 is connected to a heat-exchangerreturn riser 101, shown schematically in FIG. 8 and pictorially in FIG.9, that returns the air-assisting liquid 48 from an entire column ofheat exchangers, connected in parallel, to the chilled-liquid system.This return water has been warmed to a return temperature T_(R1) byabsorption of heat from the closed air loop 23. If the chilled-liquidsystem is a typical chilled-water system, the typical return temperatureis T_(R1)=25-27° C., such that the temperature rise T_(R1)-T_(S1) of thewater across the heat exchanger is typically within a preferable rangeof about 5-10° C., predicated on the disclosed system.

In a prototype embodiment, the heat load of the low-power-densitycomponents per front blade row 8 is experimentally about 5.4 kW to 6.9kW. This heat load is adequately cooled by one of the prototype heatexchangers 7 when it carries a flow rate of approximately 11.4 liter/min(3.0 gallon/min) of the air-assisting liquid 48.

Plumbing Connections for Quick-Connect Manifolds

Referring to the schematic FIG. 8 and the pictorial representation inFIG. 9, a front quick-connect supply riser 102 supplies thedirect-cooling liquid 49 from a second chilled-liquid system (notshown), at a supply temperature T_(S2), to M_(F) frontquick-connect-manifold supply pipes 53, one of which belongs to each ofthe M_(F) front quick-connect manifolds 9. Likewise, a rearquick-connect supply riser 103 supplies the direct-cooling liquid 49from the second chilled-liquid system, at the supply temperature T_(S2),to M_(R) rear quick-connect-manifold supply pipes 104, one of whichbelongs to each of the M_(R) rear quick-connect manifolds 14. If thesecond chilled-liquid system is a chilled-water system, as is well knownin the art, then the temperature of water in the quick-connect supplyrisers 102, 103 is typically T_(S2)=18 to 20° C. in order to be safelyabove the dew-point temperature of typical computer-room environments.

Still referring to the schematic FIG. 8 and the pictorial FIG. 9, M_(F)front quick-connect-manifold return pipes 54 (one per frontquick-connect manifold) return the direct-cooling liquid 49 to a frontquick-connect return riser 105, and thence to the return side of thesecond chilled-liquid system. Similarly, M_(R) rear quick-connectmanifold return pipes 106 (one per rear quick-connect manifold) returnthe direct-cooling liquid 49 to a rear quick-connect return riser 107,and thence to the return side of the second chilled-liquid system.Because the direct-cooling liquid has absorbed, from the front and rearblades, the heat that was dissipated by the high-power-densitycomponents therein, the return water has a temperature T_(R2) that ishigher than T_(F2). If the second chilled-liquid system is achilled-water system, as is well known in the art, then the temperatureof water in the quick-connect return risers 105 is typically T_(R2)=25to 27° C.; that is, the flow rate through the manifolds is typicallyadjusted to produce a water-temperature rise,

TηT_(R2)-T_(S2), of about 5-10° C.

Prototype Blade

One of the front blades 10 used in the prototype embodiment isillustrated in FIG. 10 and in FIG. 11. FIG. 10 shows the blade from the−x direction, whereas FIG. 11 shows the blade from the +x direction. Inorder to display the blade's internal structure, its left sheet-metalskin 65 is hidden in FIG. 10, whereas its right sheet-metal skin 66 ishidden in FIG. 11. From the description above, it is evident that theparticular structure of this blade is merely an example of the type ofequipment that may be cooled in accordance with the invention; ingeneral, the invention applies regardless of the locations of thelow-power-density and high-power-density heat-producing devices withinthe volume of the blades. Nevertheless, the blade structure describedherein has several advantages, as elucidated hereinbelow.

The blade comprises a blade circuit card 108 having a front surfacefacing the −x direction (shown in FIG. 10) and a rear surface facing the+x direction (shown in FIG. 11). At the +y edge of the blade circuitcard 108, the midplane connectors 19 are electrically connected thereto.Also electrically connected to the blade circuit card 108 are four typesof heat-producing components: first, four groups of DIMMs (“Dual-In-LineMemory Modules”), a standard format for carrying computer-memory chips,including an upper-front DIMM array 109, a lower-front DIMM array 110,an upper-rear DIMM array 111, and a lower-rear DIMM array 112; second,two processor modules, including an upper processor module 113 and alower processor module 114; third, four DIMM-power converters, includingan upper-front DIMM-power converter 115, a lower-front DIMM powerconverter 116, an upper-rear DIMM power converter 117, and a lower-rearDIMM power converter 118; and fourth, two processor-power converters,including an upper processor power converter 119 and a lower processorpower converter 120. Note that the power converters 115-118 includefined heat sinks, which are visible in FIGS. 10 & 11 and which obscurethe active electronic components used for power conversion that aremounted to circuit card 108. Of all these heat-producing components,only the processor modules (113, 114) are high-power-density,direct-liquid cooled components; all of the others arelow-power-density, air-cooled components.

Each “power-converter” component 115-120 delivers low-voltage,high-amperage power to the DIMM array 109-112 or to the processor module113, 114 that lies directly opposite on the other side of the bladecircuit card 108. For example, the upper-front DIMM power converter 115(FIG. 11) delivers power to the upper-front DIMM array 109 (FIG. 10);the lower-rear DIMM-power converter 118 delivers power to the lower-rearDIMM array 112. Likewise, the upper-processor power converter 119delivers power to the upper processor module 113, and thelower-processor power converter 120 delivers power to the lowerprocessor module 114. This arrangement, in which each power converterlies directly opposite the component it powers, produces very shortelectrical paths from the power converters to their respective loads,thereby providing a low-loss means of delivering the low-voltage,high-amperage power, and representing an advantage of this invention.

An additional advantage of this invention resides in that the DIMMarrays 109-112 are arranged such that, in traversing a blade, nostreamline of air passes through more than one DIMM array, therebypreventing overheated air that would lead to poor cooling of componentsfurthest downstream. For example (FIG. 10), on its path through theblade, an air streamline 121 passes through the lower-rear DIMM powerconverter 118 and then through the upper-front DIMM array 109. Becausethe DIMM power converter 118 dissipates only about 18% as much heat asthe lower-rear DIMM array 112 that it powers, the arrangement of DIMMsshown is, from a cooling viewpoint, far superior to an alternativearrangement having all DIMM arrays 109-112 on one side of the bladecircuit card 108 and all DIMM power converters 115-118 on the otherside, because in that case, some air streamlines would pass through twoDIMM arrays.

Both of the advantages cited above, i.e., short electrical paths forpower delivery and efficient DIMM arrangement to avoid overheated air,derive from components being placed on both sides of the blade circuitcard 108. This is possible only if the blade circuit card stands in aplane parallel to yz which lies, as shown in FIG. 12, midway between theblade's left sheet-metal skin 65 and its right sheet-metal skin 66. Thismust be done while maintaining the −z and +z faces of the blade open soas to allow for a vertical airflow. Referring to FIG. 11 and FIG. 12,these requirements are met by suspending the blade circuit card 108 andits components from an upper angle bracket 122 and a lower angle bracket123 that are attached to a tailstock 129, and from a left U-channelstrut 124 and a right U-channel strut 125 that are attached to the leftsheet-metal skin 65 and the right sheet-metal skin 66, respectively.

Referring to FIGS. 10 and 11, in the prototype embodiment, eachair-cooled DIMM array 109-112 comprises 16 double-high DIMMs 126 on11-mm centers. Prototype DIMM cards are thermal mockups, containingsimple resistors to generate heat, rather than real DRAM and hub chips.Each mockup DIMM dissipates either 20 W or 26 W of heat, depending onits configuration, so that each DIMM array 109-112 dissipates about 320W (with 20 W DMMs) or 416 W (with 26 W DMMs). Air-cooled DIMM powerconverters 115-118 and processor power-converters 119-120, whichtypically dissipate only 18% as much heat as the devices they power, arenot simulated thermally in the prototype embodiment. However, eachprototype blade has, near the midplane connectors 19, additionalair-cooled heat-producing components (not shown in the Figures) thatdissipate 80 W. Thus the total air-cooled heat dissipation per prototypeblade is (4)(320)+80=1360 W (with 20 W DIMMs) or (4)(416)+80=1744 W(with 26 W DIMMs). In order to measure the thermal performance of theprototype embodiment, over 11,000 temperature sensors are located nearheat-producing components on the mockup DIMM cards throughout theprototype enclosure 1. With 1360 air-cooled watts per blade, and waterentering the heat exchangers 7 at 15° C. (slightly lower than the range18-20° C. suggested hereinabove, because humidity in the prototype'slaboratory environment is controlled such that 15° C. is well above dewpoint), the highest temperature measured by these sensors is 56° C. With1744 air-cooled watts per blade, the highest air-cooled-componenttemperature measured is 71° C. The highest temperatures are typicallylocated near the downstream edges of DIMM cards, where ambient air iswarmest due to heating thereof by components upstream.

In the prototype embodiment, each mockup processor module contains an18.5×18.5 mm silicon heater chip dissipating 350 W of heat, yielding anaverage power density of 1.02 W/mm². This heat is removed by thedirect-cooling liquid 49 that flows in a processor cooling head 127(FIG. 10). For the prototype embodiment, the direct-cooling liquid iswater, which enters the cooling head 127 at 10° C. (lower than the range18-20° C. suggested hereinabove, because humidity in the prototype'slaboratory environment is controlled such that 10° C. is above dewpoint). The prototype cooling head, a silicon-microchannel cooler, isattached to the silicon chip in such a way as to remove the chip's heatefficiently, in order to maintain the chip at the lowest possibletemperature. Thermal sensors integrated into the silicon heater chipillustrate that, with water flowing at 1 liter/min through the coolinghead, the maximum temperature on the silicon chip is about 35° C. Thisrepresents a total thermal resistance (chip to inlet water) of 0.07°C./W. Using the chip area given above, this is equivalent to anarea-normalized thermal resistance of 24° C./(W/mm²). In an alternativeembodiment, the prototype blades are populated with heater chipsgenerating 96 W (0.28 W/mm) that are air cooled by Heatlane™ heatsinktechnology. This air-cooled solution provides a total thermal resistanceof 0.27° C./W, which is equivalent to an area-normalized resistance of92° C./(W/mm²). Thus, the direct-water-cooled solution has nearly fourtimes the cooling capability of the air-cooled solution.

Referring to FIG. 10, the direct-cooling liquid 49 which cools theprocessor modules enters the blade through the supply hose 73 andblade-supply elbow fitting 74, as previously described, thereafter to afeed-through supply fitting 128 that passes through a hole in the bladetailstock 129, then to a blade supply pipe 130, then to a flow meter 131which verifies that an adequate flow of direct-cooling liquid is presentbefore power is applied to the processor modules 113-114 and finally toa blade-supply manifold 132 that feeds two hoses, including anupper-processor supply hose 133 and a lower-processor supply hose 134,which convey the direct-cooling liquid 49 to the cooling heads 127 thatcool the upper processor module 113 and lower processor module 114.After passing through the cooling heads 127, the direct-cooling liquidflows through an upper-processor return hose 135 and a lower-processorreturn hose 136, whose flows are combined in a blade-return manifold137. Referring now to FIG. 11, the blade-return manifold 137 dischargesthis flow to a return elbow fitting 138 that passes through a hole inthe blade circuit card 108 and delivers the flow to a blade return pipe139, from there to a return feed-through fitting 140 that passes througha hole in the blade tailstock 129, then to the blade return elbowfitting 75, and finally to the return hose 76.

Air Movers

The air-moving assemblies 34, 39 are now described in more specificdetail, along with related issues such as acoustic insulation, sealing,flow control, and fan failure.

Each of the fans 35-38 and 40-43 driving the closed-loop airflow 23 ispreferably of the type known as a “centrifugal fan” or “blower”, becausesuch fans naturally cause the air to turn a right-angle corner. Thus, ifthe upper fans 35-38 are of the centrifugal type, they naturally causethe air to turn at the upper-front airflow corner 30 (FIG. 1);similarly, if the lower fans 40-43 are centrifugal, they naturally causethe air to turn at the lower-rear airflow corner 32.

Referring to FIG. 13 and assuming the use of centrifugal fans, each ofthe upper fans 35-38 has an axis of rotation that is parallel to thez-axis, an intake air-stream 141 flowing toward +z, and an exhaustair-stream flowing 142 flowing toward +y. The latter flow direction isachieved by a fan-and-housing assembly 143, as shown in FIG. 13, whereineach upper fan 35-38 is enclosed in a housing 144 having two open sides;i.e., an intake side 145 facing −z and an exhaust side 146 facing +y.Similarly, each of the lower fans 40-43 has an axis of rotation that isextended in parallel with the z axis, an air-intake direction pointingtoward −z, and an air-exhaust direction pointing toward −y. The latteris achieved (assuming that upper and lower fans are identical) by thefan-and-housing assembly 144, which for the lower fans is oriented sothat the open intake side 145 faces −z and the open exhaust side 146faces −y. In other words, if lower fans and upper fans are identical,then a lower fan 40-43 in its housing 144 is merely an upside downversion of an upper fan 35-38 in its housing 144.

Because the air loop 23 is closed, the noise created by the moving air,and in particular the noise created by the fans 35-38 and 40-43, may beacoustically isolated inside the enclosure 1, thereby minimizingannoyance to nearby personnel, and protecting their hearing. Incontrast, acoustical isolation is much more difficult to achieve forconventional enclosures where the air used for air cooling therewithinflows across the enclosure boundary to the outside. Improved acousticisolation is thus a key advantage of the present invention. Acousticisolation of the fan and air noise within the enclosure 1 is readilyaccomplished by lining all inside surfaces of its outer shell,especially walls and doors 2, with a layer 147 of an acousticinsulation, as shown in FIG. 3. This insulation should preferably be ofthe type known as a “transmission-loss material”, which attenuates thetransmission of acoustic energy therethrough. A transmission-lossmaterial is primarily characterized by its mass, the greater the massper unit area of the layer, the greater the attenuation. In a prototypeembodiment of this invention, the acoustic insulation used was a1″-thick (25.4-mm-thick) layer of SoundMat PB, which is a self-adhesivetransmission-loss material, made by SoundCoat corporation, that includesa “barrier layer” (transmission-loss layer) having an areal density of 1lb/ft² (4.88 N/m²).

Referring again to FIGS. 1, 2A and 2B, the top-rear region 46 {y>0;z>z₂} of the enclosure 1 contains only air, thereby providing ahigh-pressure plenum in which the closed air loop 23 turns theupper-rear airflow corner 31, the air being driven to execute this turnbecause of the favorable pressure gradient created in the −z directionby the low-pressure intake of the lower fans 40-43. Likewise, abottom-front region 47 of the enclosure 10 contains only air, therebyproviding a high-pressure plenum in which the closed air loop 23 turnsthe lower-front airflow corner 33, the air being driven to execute thisturn because of the favorable pressure gradient created in the +zdirection by the low-pressure intake of the upper fans 35-38.

The fans in each air-moving assembly (34, 39) are arranged so that theirair streams do not substantially interfere. This is achieved by placingthe fans in an over-and-under, fore-and-aft arrangement shown in FIG.14, which depicts, without fan housings 144, the upper fans (35-38) ofthe upper air-moving assembly 34. Because of the fore-and aftarrangement, outer-intake airstreams 150 and 151 of the outer fans (35,36) do not interfere with inner-intake airstreams 152 and 153 of theinner fans (37, 38). Also, because of the over-and-under arrangement,outer-exhaust airstreams 154 and 155 of the outer fans (35, 36) do notinterfere and with inner-exhaust airstreams 156 and 157 of the innerfans (37, 38).

In the prototype embodiment, all fans (upper fans 35-38 and lower fans40-43) are preferably ebm/papst model R3G250, backward-curvedcentrifugal fans having a 250-mm-diameter rotating wheel 158 comprisingeleven backward-curved blades 159. The pressure-flow performance of eachsuch fan is enhanced with a flared inlet ring 160, which guides airsmoothly into the fan. When such an inlet ring is used, space in the zdirection may be saved, as shown, by packaging the fans so that thewheel 158 of each inner fan 37, 38 partially overlaps (in the zdirection) the inlet ring 160 of the corresponding outer fan 35, 36. Therotating wheel 158 is attached to the armature of a motor whose stator161 is affixed to the housing 144.

FIG. 15 shows four of the fan-and-housing assemblies 143 arranged in theaforesaid over-and-under, fore-and-aft configuration. Each housing 144comprises a sheet-metal box 162 whose −z and +y faces are open (as shownpreviously in FIG. 13), a top-hat-shaped flange 163 into which thestator 161 nestles and to which it is affixed, and a connector assembly164 on whose +y face is located a male fan connector 165 that provides,via a local fan cable 166, for connection of electrical power to thefan's motor, as well as connection of electrical signals from the fan'stachometer.

FIG. 16 shows the upper air-moving assembly 34, which in the prototypeembodiment comprises, in addition to the four fan-and-housing assemblies40, a sheet-metal four-fan enclosure 167 whose surfaces facing the −zand +y directions are substantially open, as shown in FIG. 17, so as tominimize aerodynamic resistance that would impede intake airstreams 150,152 and exhaust airstreams 154, 156. Referring again to FIG. 16, theupper air-moving assembly 34 also comprises an array of four female fanconnectors 168 that are attached to the four-fan enclosure 167 and whichmate with the male-fan connectors 165 on the fan-and-housing assemblies143 in order to provide the fans with electrical power and signals thatoriginate from remote equipment (not shown), and are carried to thefemale fan connectors by remote fan cables 169, which are retained andprotected by surrounding skids 170. A similar design pertains to thelower air-moving assembly 39, which is just an upside-down replica ofthe upper air-moving assembly 34.

A handle 171 is provided on each fan-and-housing assembly 143 tofacilitate its removal from the upper air-moving assembly 34 should afan fail. Removal of one of the outer fans 35, 36, which automaticallydisconnects its male fan connector 165 from the female fan connector168, is accomplished by releasing a latch 172 and pulling the handle 171in the −y direction, as illustrated with regard to the outer right fan36 in FIG. 18. This figure also reveals a support box 173. There are twosuch boxes, as shown, whose function is to hold the outer fans 35, 36 inposition against the force of gravity while still allowing unimpededflow of air in the z direction. In the prototype embodiment, removingone of the outer fans 35, 36 (and one of the support boxes 173) isprerequisite to removing one of the inner fans 37, 38; however, this isnecessary only if the outer and inner fans overlap in the z direction tosave space (as previously described in connection with FIG. 14). Ifthere is no z overlap, then the outer and inner fans may be madeindependently removable, like conventional drawers.

It is important that the closed air loop 23 be reasonably tightly sealedto ensure that air will not leak from high-pressure areas tolow-pressure areas, because such leaks would diminish the amount ofcooling air that actually circulates in the closed air loop 23. Sealingis particularly important in the vicinity of the top-rear-region 148 andbottom-front region 149, because these areas are high-pressure plenumsthat readily leak air to the atmospheric pressure surrounding theenclosure 1 and to other low-pressure regions such as the fan intakes.For example, in the prototype embodiment, the positive pressure in theplenums is approximately 280 to 350 Pa (depending on conditions), andthe pressure difference across the stack of front thermally neutralunits is about 400 Pa. To prevent depressurization of the high-pressureplenums when the doors 2 of the enclosure are opened for access, aplenum box 174, which is open on its −y and −z faces, encloses thetop-rear region 148, and another such box (shown in FIG. 19), which isopen only on its +y and +z faces, encloses the bottom-front region 149.In addition, in order to prevent aerodynamic short-circuiting of thefans, it is important that the high-pressure plenums be sealed off fromthe low-pressure fan intakes, by placing sealing means between the +zsurface of the lower fan unit 39 and +y surface of the lowest frontblade cage 50.

With the prototype embodiment, it has been ascertained experimentallythat a splitter plate 175 shown in FIG. 19, which bisects the plenum box174 parallel to the yz plane, aids in distributing the air emerging fromthe lower fans more evenly in the plenum box, thereby sharing the airmore evenly between the +x and −x halves of the enclosure 1 and leadingto better thermal performance (lower maximum temperature) of air-cooledheat-producing components arranged in the enclosure. Without thesplitter plate 175, the air emerging from the fans tends to favor the −xside of the plenum box 174, apparently due to the clockwise rotation ofthe fans in FIG. 19. The splitter plate 175 helps prevent too much airfrom flowing to the −x side of the plenum box by guiding the flow of airfrom the right-side fans to stay in the right half of the plenum box.The maximum air-cooled temperature in the prototype enclosure 1 wasthereby reduced by about 6° C.

FIG. 20 shows the topmost thermally neutral front unit 28 as well as theupper air-moving assembly 34. To allow for failure of one of the fans35-38 without overheating the air-cooled heat-producing components 21,the preferred embodiment of the invention has a separation S between the+z face of the blade cage 50 and the −z a face of the four-fan enclosure167. It must be noted that when a fan such as 38 fails, in order toprevent aerodynamic short-circuiting of the other fans through the openaperture created by the failed fan, it is necessary that thefailed-fan's exhaust area ABCD be sealed by a pivoting flat plate (notshown) that is hinged along BC and normally blown open by the airstream157, but which falls under the force of gravity or a spring when thereis a cessation in the failed-fan's exhaust airstream 157. Thus, the airthat normally travels upward through the failed fan (airstream 153) mustbe imparted an alternative route; otherwise, heat-producing componentsdirectly beneath the failed fan, in the blade cage 50, will overheat.This alternative route is provided for by the separation S, which allowsalternative air paths such as 176 through the other, still-functioningfans. Let

T be the increase in worst-case temperature of heat-producing componentsdue to a fan failure. In the prototype embodiment, experiments show

T=32° C. with S=10 mm (the smallest value of S tested), but that

T=2° C. with S=35 mm. That is, with S=35 mm, the system and functioningthereof is virtually immune to the failure of a fan.

Referring to FIG. 21, alternative embodiments may use other techniquesto allow for fan failure without sacrificing the vertical spacerepresented by S. For example, even with S=0, an alternative path forthe airstream normally handled by one of the outer fan 35, 36 may beprovided by permanent openings 177 in the side walls of the supportboxes 173 that otherwise separate the intake airstreams of the two outerfans. Then, for example, if fan 36 fails, an alternative air path suchas 178 is possible, which prevents heat-producing components beneath thefailed fan from overheating. If one of the inner fans 37, 38 fails,providing an alternative air path with S=0 is more difficult than forthe outer fans, because, for example, the wall 179 between inner fan 38and outer fan 36 seals the high-pressure exhaust of fan 38 from thelow-pressure intake of fan 36, and thus must not pass air under normalcircumstances when all fans are running. However, it is possible toconstruct a louvered opening in wall 179, whose louvers would open onlywhen fan 38 fails.

Bulk Power Supply

FIG. 22 illustrates the manner in which bulk power supplies, whichconvert electrical power from AC to DC, may be integrated into theenclosure 1 described by this invention. Such supplies, which ordinarilyare 90% efficient and thus dissipate internally as heat about one-ninth(11%) of the power they deliver, are needed for typical heat-producingcomponents, such as computer processor and memory chips. Off-the-shelfbulk-power-supply modules 180 are readily available that provide 4 kW ofpower in a package having xyz dimensions of about 127 mm×385 mm×127 mm.FIG. 22 illustrates one embodiment of the invention in which twenty-foursuch power-supply modules (enough for an 80 kW rack with redundancy) arehoused in their own power-supply enclosure 181 that abuts the −x face ofthe enclosure frame 3. The power-supply enclosure 181 is aerodynamicallyisolated from the main enclosure 1 so that the closed loop of air 23 inthe main enclosure 1 does not enter the power-supply enclosure 181. Thepower-supply modules 180 are arranged in two stacks; i.e., a front stack182 whose modules 180 are removable (for repair and replacement) fromthe −y face of the power-supply enclosure 181; and a rear stack 183whose modules 180 are similarly removable from the +y face of thepower-supply enclosure. As shown, the front stack draws inlet coolingair 184 towards the +y direction; the rear stack draws inlet cooling air185 toward the −y direction. Both stacks 182, 183 exhaust cooling air186 towards the +z direction through an aperture 187 in the power-supplyenclosure. Thus, for such off-the-shelf power-supply modules 180, therelatively small mount of heat dissipated in the power supplies isexpelled to room air, and must be treated conventionally by externalair-handling units. For example, if the power required in the enclosure1 is 80 kW and the power supplies are 90% efficient, 8.9 kW is expelledto room air.

An alternative embodiment may employ custom bulk power-supply modulesthat admit bottom-to-top airflow, thereby allowing the power-supplymodules to employ, lice the blade cages 10, 15, closed-loopliquid-assisted air cooling, in keeping with the inventive objective ofeliminating the air-cooling burden on the machine room.

While the present invention has been particularly shown and describedwith respect to preferred embodiments thereof, it will be understood bythose skilled in the art that the foregoing and other changes in formsand details may be made without departing from the scope and spirit ofthe present invention. It is therefore intended that the presentinvention not be limited to the exact forms and details described andillustrated, but fall within the scope of the appended claims.

1. An apparatus for cooling heat-generating high-power electroniccomponents and low-power electronic components, which are housed in anenclosure, said apparatus comprising: an arrangement for cooling saidhigh-power electronic components by a direct liquid cooling circulationloop; and a further arrangement for cooling said low-power electroniccomponents by a liquid-assisted air circulation.
 2. An apparatus asclaimed in claim 1, wherein said liquid-assisted air cooling circuitcomprises a plurality of heating and cooling zones.
 3. An apparatus asclaimed in claim 2, wherein said liquid-assisted air cooling circulationloop includes a closed-loop air flow path.
 4. An apparatus as claimed inclaim 2, wherein liquids employed in said liquid-assisted air coolingcirculation loop and in said direct liquid cooling circulation loop areseparate and distinct from each other throughout their respective flowpaths.
 5. An apparatus as claimed in claim 2, wherein liquids employedin said liquid-assisted air cooling circulation stream along a sharedsegment of their flow paths.
 6. An apparatus as claimed in claim 3,wherein at least one heat exchanger is arranged in said closed-loop airflow path for transferring heat therefrom to an air-assisting liquidcooling circuit.
 7. An apparatus as claimed in claim 2, wherein airmovers are located in the air flow path of said air cooling circulationloop, said air movers each comprising centrifugal fans, which arearranged to prevent interference with their respective air-streamsflowing through said path.
 8. An apparatus as claimed in claim 7,wherein said centrifugal fans are arranged in an over-and-under,fore-and-aft, orientation in, respectively, upper and lower housings forturning the flow of air in said air circulation loop.
 9. An apparatus asclaimed in claim 8, wherein interior surfaces of said enclosure areequipped with acoustic insulation so as to acoustically attenuate theamount of acoustical noise, produced by said centrifugal fans, that istransmitted across said surfaces to the outside of said enclosure. 10.An apparatus as claimed in claim 9, wherein said air movers includemeans for diverting the flow of air streaming through said aircirculation loop into an alternate path upon failure of a centrifugalfan so as to inhibit an aerodynamic short-circuiting of the remainingcentrifugal fans.
 11. An apparatus as claimed in claim 10, wherein saiddiverting means comprises a pivotable plate for closing an exhaust ofthe failed centrifugal fan, and further comprises openable louversformed in a wall between centrifugal fans in each respective housing.12. An apparatus as claimed in claim 6, wherein quick-connects, whichare located at the same vertical space as said heat exchanger facilitateflow of direct cooling liquid through a manifold supply pipe to saiddirect liquid cooling circulation loop and therefrom to a return line soas to form a vertical space-saving configuration in said enclosure. 13.An apparatus as claimed in claim 10, wherein one or more splitter platesbisect a plenum box in said air flow so as to aid air emerging from saidlower centrifugal fans more evenly in said plenum box and improve uponcooling of the air-cooled heat generating electronic components in saidenclosure.
 14. An apparatus as claimed in claim 2, wherein saidlow-power components include a plurality of Dual-In-Line Memory Modules(DIMM), said DIMMs being arranged such that a flow of cooling air passesthrough no more than one DIMM in each respective cooling zone at anytime so as to prevent overheating of successive DIMMs by overheated airflows.
 15. An apparatus as claimed in claim 6, wherein said at least oneair-to-liquid heat exchanger is located interleaved with rows ofpackages for cooling the flow of air in each said row passage prior tosaid flow of air entering a subsequent row passage, said packagescomprising blades mounting said heat-generating components and diverseoperative components, said rows of blades being attached to at least oneside of at least one or more midplanes comprising circuit cards forelectrical interconnections.
 16. A method as claimed in claim 15,wherein, referring to an imaginary Cartesian coordinate system havingaxes x, y, and z, a first stack of said rows of blades and heatexchangers is stacked along the z axis on a −y side of said one or moremidplanes that lie in a central plane parallel to the x and z axes,similarly stacking a second stack of said rows of blades and heatexchangers along the z axis on a +y side of said one or more midplanes,a first set of air movers is located at a +z end of said first stack, afirst plenum at a +z end of said second stack, locating a second set ofair movers at a −z end of said second stack, and a second plenum islocated at a −z end of said first stack, such that said flow of air isconveyed, by means of said air movers, along a closed loop comprising,in stream-wise order, said first stack of blades through which air flowstoward the +z direction, said first set of air movers into which airflows toward the +z direction and from which it exhausts toward the +ydirection, said first plenum into which air flows toward the +ydirection and from which it exhausts toward the −z direction, saidsecond stack of blades through which air flows toward the −z direction,said second set of air movers into which air flows toward the −zdirection and from which it exhausts toward the −y direction, andfinally said second plenum into which air flows toward the −y directionand from which it exhausts toward the +z direction into said firststack, thereby completing said closed loop.
 17. An apparatus as claimedin claim 16, wherein in a blade, the electronic components are mountedon a blade circuit card and further the corresponding power convertersfor one or more electronic components are mounted on the directlyopposite surface of said blade circuit card.
 18. An enclosure forhousing heat-generating electronic components in a thermally controlledinterior, comprising: an enclosure shell; a first cooling fluid; meansfor circulating the first cooling fluid along a closed first path withinthe enclosure shell, said means including a plurality of centrifugalfans located along said closed first path; a plurality ofheat-generating regions containing the heat-generating electroniccomponents; a plurality of heat exchangers capable of transferring heatfrom the first cooling fluid to a second cooling fluid; and means forcirculating the second cooling fluid through the heat exchangers and outof the enclosure shell along a second path; wherein the first coolingfluid is alternately heated in the heat-generating regions and cooled bythe heat exchangers a plurality of times as the first cooling fluidtraverses the first closed path.
 19. An enclosure as claimed in claim18, wherein said first cooling liquid is air and air movers are locatedin the air flow path of said air cooling circulation loop, said airmovers each comprising said centrifugal fans, which are arranged toprevent interference with their respective air-streams flowing throughsaid path.
 20. An enclosure as claimed in claim 19, wherein saidcentrifugal fans are arranged in an over-and-under, fore-and-aft,orientation in, respectively, upper and lower housings for turning theflow of air in said air circulation loop.
 21. An enclosure as claimed inclaim 19, wherein interior surfaces of said enclosure are equipped withacoustic insulation so as to acoustically attenuate the amount ofacoustical noise, produced by said centrifugal fans, that is transmittedacross said surfaces to the outside of said enclosure.
 22. An enclosureas claimed in claim 19, wherein said air movers include means fordiverting the flow of air streaming through said air circulation loopinto an alternate path upon failure of a centrifugal fan so as toinhibit an aerodynamic short-circuiting of the remaining centrifugalfans.
 23. An enclosure as claimed in claim 22, wherein said divertingmeans comprises a pivotable plate for closing an exhaust of the failedcentrifugal fan, and further comprises openable louvers formed in a wallbetween centrifugal fans in each respective housing.
 24. A method ofcooling heat-generating high-power electronic components and low-powerelectronic components, which are housed in an enclosure, said methodcomprising: cooling said high-power electronic components by a directliquid cooling circulation loop; and cooling said low-power electroniccomponents by a liquid-assisted air circulation.
 25. A method as claimedin claim 24, wherein said liquid-assisted air cooling circuit comprisesa plurality of heating and cooling zones.
 26. A method as claimed inclaim 25, wherein said liquid-assisted air cooling circulation loopincludes a closed-loop air flow path.
 27. A method as claimed in claim26, wherein liquids employed in said liquid-assisted air coolingcirculation loop and in said direct liquid cooling circulation loop areseparate and distinct from each other throughout their respective flowpaths.
 28. A method as claimed in claim 27, wherein at least one heatexchanger is arranged in said closed-loop air flow path for transferringheat therefrom to an air-assisting liquid cooling circuit.
 29. A methodof cooling heat-generating electronic components in a thermallycontrolled interior of an enclosure shell, said method comprising:circulating a first cooling fluid along a closed first path within theenclosure shell, locating a plurality of centrifugal fans along saidclosed first path, said path having a plurality of heat-generatingregions containing the heat-generating electronic components; having aplurality of heat exchangers transferring heat from the first coolingfluid to a second cooling fluid; and circulating the second coolingfluid through the heat exchangers and out of the enclosure shell along asecond path; wherein the first cooling fluid is alternately heated inthe heat-generating regions and cooled by the heat exchangers aplurality of times as the first cooling fluid traverses the first closedpath.
 30. A method as claimed in claim 29, wherein said first coolingfluid is air, locating air movers in the air flow path of said aircooling circulation loop, said air movers each comprising saidcentrifugal fans, which are arranged to prevent interference with theirrespective air-streams flowing through said path.
 31. A method asclaimed in claim 30, wherein said air-to-liquid heat exchangers areinterleaved with rows of packages for cooling the flow of air in eachsaid row passage prior to said flow of air entering a subsequent rowpassage, said packages comprising blades mounting said heat-generatingcomponents and diverse operative components, said rows of blades beingattached to at least one side of at least one or more midplanescomprising circuit cards for electrical interconnections.
 32. A methodas claimed in claim 31, wherein, referring to an imaginary Cartesiancoordinate system having axes x, y, and z, stacking a first stack ofsaid rows of blades and heat exchangers along the z axis on a −y side ofsaid one or more midplanes that lie in a central plane parallel to the xand z axes, similarly stacking a second stack of said rows of blades andheat exchangers along the z axis on a +y side of said one or moremidplanes, a first set of air movers is located at a +z end of saidfirst stack, locating a first plenum at a +z end of said second stack,locating a second set of air movers at a −z end of said second stack,and locating a second plenum at a −z end of said first stack, such thatsaid flow of air is conveyed, by means of said air movers, along aclosed loop comprising, in stream-wise order, said first stack of bladesthrough which air flows toward the +z direction, said first set of airmovers into which air flows toward the +z direction and from which itexhausts toward the +y direction, said first plenum into which air flowstoward the +y direction and from which it exhausts toward the −zdirection, said second stack of blades through which air flows towardthe −z direction, said second set of air movers into which air flowstoward the −z direction and from which it exhausts toward the −ydirection, and finally said second plenum into which air flows towardthe −y direction and from which it exhausts toward the +z direction intosaid first stack, thereby completing said closed loop.
 33. A method asclaimed in claim 32, wherein in a blade, the electronic components aremounted on a blade circuit card and further the corresponding powerconverters for one or more electronic components are mounted on thedirectly opposite surface of said blade circuit card.
 34. A method asclaimed in claim 30, wherein said centrifugal fans are arranged in anover-and-under, fore-and-aft, orientation in, respectively, upper andlower housings for turning the flow of air in said air circulation loop.35. A method as claimed in claim 29, wherein interior surfaces in saidenclosure are equipped with acoustic insulation so as to attenuate theamount of acoustical noise, produced by said centrifugal fans, that istransmitted across said surfaces to the outside of said enclosure.
 36. Amethod as claimed in claim 30, wherein said air movers divert the flowof air streaming through said air circulation loop into an alternatepath upon failure of a centrifugal fan so as to inhibit an aerodynamicshort-circuiting of the remaining centrifugal fans.